Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] Alerts Treemap and Multi-Dimensional Alert Grouping #126896

Merged
merged 8 commits into from
Jul 13, 2022

Conversation

andrew-goldstein
Copy link
Contributor

@andrew-goldstein andrew-goldstein commented Mar 4, 2022

[Security Solution] Alerts Treemap and Multi-Dimensional Alert Grouping

This PR introduces the new Treemap and Multi-Dimensional Alert Grouping to the Alerts page.

The initial commit was developed as an ON week proof of concept (POC). It has since been updated to incorporate product and UX feedback.

Alerts treemap

The new Alerts page treemap is shown in the screenshot below:

treemap

Above: The new treemap in the Alerts page

  • Alerts are colored by risk score
  • Clicking on a cell instantly filters the Alerts page
  • Treemap legend items may be added to filters and Timeline investigations
  • The new treemap supports multi-dimensional grouping and filtering
    • Alerts are grouped by kibana.alert.rule.name and host.name by default

Multi-Dimensional Alert Grouping

The table on the Alerts page, which previously supported grouping by a single field, has also been enhanced to support multi-dimensional grouping, per the screenshot below:

alerts-table-multi-dimensional-grouping

Above: The table in the Alerts page, enhanced to support multi-dimensional grouping

Filtering the Alerts page by risk score

Every rule, including prebuilt Elastic rules and custom rules created by users, must specify a risk score at rule creation time, per the screenshot below:

rule_risk_score_configuration

Above: Every rule has a risk score specified when it's created

The colors of the alerts displayed in the treemap are determined by the rule's risk score. This makes it easy to quickly filter the entire alerts page by clicking on the riskiest alerts.

Clicking on a cell in the treemap adds two filters, one for each group by field, per the screenshot below:

two-filters

Above: Two filters, (one for each group by field), are added to the page when a cell is clicked

The Alerts page updates instantly when filters are added or removed. In the screenshot below, the 2nd filter was removed to filter the page to all mimikatz process started alerts:

second-filter-removed

Above: Removing the 2nd filter, a specific host.name, revealed all the hosts in the mimikatz process started alerts

Switching views

Users may switch between the following views:

  • Table
  • Trend (default)
  • Treemap

per the screenshot below:

view-selection

Above: View selection

The default Trend view is shown in the screenshot below:

trend-view

Above: The (default) Trend view

tooltip

Above: The Trend view only supports visualizing a single dimension

Collapsing the panel

The panel may optionally be collapsed to save space, per the screenshot below:

collapsed

Above: The panel (optionally) collapsed

User preferences are persisted to local storage

Previously, the Group by selections on the Alerts page were always forgotten when users navigated away from the Alerts page. As a result, users had to re-select their preferred Group by fields every time they visited the page.

We now store all of the following preferences in local storage:

  • View selection (Table, Trend, Treemap)
  • Group by selections
  • Panel collapse state

The preferences above are now restored the next time users return to the Alerts page.

Group by field selection is synchronized between views

Group by field selection is synchronized between views. For example, if a user changes the Group by fields in the Treemap view and then switches to the Table view, the same Group by fields will be displayed in the Table view.

Resetting Group by fields to their defaults

Users may reset the Group by fields to their defaults (kibana.alert.rule.name and host.name) for any visualization via the menu shown in the screenshot below:

reset-group-by-fields

Above: Resetting Group by fields to defaults via the menu

@andrew-goldstein andrew-goldstein added release_note:enhancement WIP Work in progress Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. auto-backport Deprecated - use backport:version if exact versions are needed Team:Threat Hunting:Investigations Security Solution Investigations Team v8.2.0 labels Mar 4, 2022
@andrew-goldstein andrew-goldstein self-assigned this Mar 4, 2022
@andrew-goldstein andrew-goldstein changed the title [Security Solution] Alerts by risk score treemap [DRAFT] [Security Solution] Alerts by risk score treemap Mar 4, 2022
@andrew-goldstein
Copy link
Contributor Author

This draft PR is ready for discussion, but not code review

@andrew-goldstein andrew-goldstein force-pushed the alerts-treemap branch 5 times, most recently from 3e83243 to 74967a3 Compare March 11, 2022 08:39
@andrew-goldstein andrew-goldstein force-pushed the alerts-treemap branch 2 times, most recently from d075678 to 9bf04dd Compare May 5, 2022 07:57
@andrew-goldstein
Copy link
Contributor Author

Archived: Original POC design

For reference, this comment archives the original ON week POC design from the initial commit, when the PR was opened as a [DRAFT]:

[Security Solution] Alerts by risk score treemap

This PR introduces the Alerts by risk score treemap to the Alerts page:

alerts_by_risk_score_treemap

Above: The new Alerts by risk score treemap

  • Alerts are colored by risk score
  • Clicking on a cell instantly filters the alerts page
  • Users may specify custom fields to group by, or quick-select from our curated sets of fields that pair well together

Feature flag

The features in this PR are enabled via the following feature flag in experimental_features.ts:

  alertsTreemapEnabled: true,

When the feature flag is disabled, the alerts page looks like main today:

feature_flag_disabled

Above: This branch, when the feature flag is disabled

Let's take an (EUI) tour

When the feature flag is enabled, the layout of the Alerts page incorporates the design detailed in #120287, which eliminates unusable space at the top of the Alerts page, and enables users to quickly hide and show the charts.

Hiding and showing charts

Users are introduced to the new Hide charts feature of the layout via Step 1 of an EUI Tour:

tour_step_1

Above: Step 1 of the tour introduces the Hide charts feature

The tour is interactive

Users may (optionally) try the Hide charts feature as part of the tour, per the video below:

hide_show_chart.mov

Above: (video) The interactive tour let's users experience the new Hide charts feature

Switching views

Step 2, the final step of the tour, introduces the view selector:

tour_step_2

Above: Step 2 of the tour introduces the view selector

Per the screenshot and video below, view selection toggles between the default Trend view and the new Alerts by risk score view:

view_selection

Above: View selection

view_selection.mov

Above: (video) View selection toggles between the Trend view and the Alerts by risk score view

Preferences are persisted to local storage

We store the user's chart visibility and view preferences in local storage.

When users navigate to other views in Kibana or close the browser, their chart visibility and view preferences are restored the next time they visit the Alerts page.

Filtering the Alerts page by risk score

Every rule, including prebuilt Elastic rules and custom rules created by users, must specify a risk score at rule creation time, per the screenshot below:

rule_risk_score_configuration

Above: Every rule has a risk score specified when it's created

The colors of the alerts displayed in the treemap are determined by the rule's risk score.

This makes it easy to quickly filter the entire alerts page by clicking on the riskiest alerts, per the video below:

filtering_via_the_treemap.mov

Above: (video) Users click on the riskiest alerts to filter the page

Clicking on a cell in the treemap adds two filters, one for each group by field:

two_filters

Above: Two filters, (one for each group by field), are added to the page when a cell is clicked

In the screenshot below, the 2nd filter was removed to reveal all the hosts in mimikatz process started alerts:

2nd_filter_removed

Above: Removing the 2nd filter, host.name, revealed all the hosts in the mimikatz process started alerts

Instant insights

We can instantly answer questions like "Did this host trigger alerts for other rules?", and at a glance, grok the severity of those alerts, by removing the first of the two filters:

1st_filter_removed

Above: Removing the first filter, kibana.alert.rule.name, revealed other alerts (from different rules) triggered for a specific host

Custom group by criteria

Users may customize the group by criteria:

custom_group_by.mov

Above: (video) Entering custom group by criteria

Persistance via local storage

When users navigate to other views in Kibana or close the browser, the previously-selected group by fields are restored from local storage the next time they visit the Alerts page.

Restoring defaults

Clicking on the first menu item in the settings gear, Reset group by fields, restores the default group by fields, per the video below:

restoring_default_groups.mov

Above: (video) Restoring the default group by fields

Quick-select from Elastic-curated group by criteria

Users may quick-select from Elastic-curated sets of fields that pair well together, like parent and child processes, or process and file name:

quick_select_groups

Above: Quick-select from Elastic-curated sets of fiels

A Multi-dimensional Count table

Previous versions of the Count table were limited to grouping by one field. This PR enhances it to support grouping by two fields, per the screenshot below:

count_two_group_by_fields

Above: The Count table now supports two group by fields

Quickly toggle between table and tree views

The group by criteria of the Count and Alerts by risk score widgets are linked together, making it easy to toggle between the two views, per the video below:

linked_group_by_criteria.mov

Above: (video) The Count and Alerts by risk score group by criteria are linked

Trend chart legend enhancements

The Trend chart's legend has been enhanced to display counts, per the design detailed in issue #120282

counts_in_trend_widget_legend

Above: The Trend chart legend, enhanced to display counts

Note: The original intention of adding counts to the Trend chart legend, as documented in #120282, was to combine the Trend and Count widgets into a single view via the legend.

In this PR, the Trend and Count widget queries have diverged; (the latter now supports multiple group by fields). This prevents the Trend and Count widgets from being combined into a single view.

Trend chart field selection is persisted to local storage

This PR adds persistence of the Trend chart Stack by field to local storage.

When users navigate to other views in Kibana or close the browser, the previously-selected field is restored the next time they visit the Alerts page.

@andrew-goldstein
Copy link
Contributor Author

Archived: Recommended groups POC

In this release, users must manually select Group by fields that pair well together, e.g. parent and child processes, or process name and file name.

For reference, this comment archives the Recommended groups POC, where we may proffer curated groups that pair well together, demonstrated in the video below:

recommended_groups.mov

Above: A video demonstrating the Recommended groups (POC)

@andrew-goldstein
Copy link
Contributor Author

Archived: Color palette selection POC

In this release, the colors of the alerts displayed in the treemap are determined by the rule's risk score.

For reference, this comment archives the Color palette selection POC, demonstrated in the video below:

color_palettes.mov

Above: A video demonstrating the Color palette selection (POC)

@andrew-goldstein andrew-goldstein added v8.4.0 and removed WIP Work in progress v8.2.0 labels Jul 11, 2022
@andrew-goldstein andrew-goldstein changed the title [DRAFT] [Security Solution] Alerts by risk score treemap [Security Solution] Alerts Treemap and Multi-Dimensional Alert Grouping Jul 11, 2022
@andrew-goldstein andrew-goldstein marked this pull request as ready for review July 11, 2022 12:20
@andrew-goldstein andrew-goldstein requested review from a team as code owners July 11, 2022 12:20
export const RISK_COLOR_HIGH = euiLightVars.euiColorVis7;
export const RISK_COLOR_CRITICAL = euiLightVars.euiColorVis9;

export const RISK_SCORE_LOW = 21;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a comment just detailing why the values are these. We may want to reference this for the docs team as well

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a comment just detailing why the values are these. We may want to reference this for the docs team as well

😂 I didn't define or change these values, so I can't speak to, or document why for example low is 21.

This PR merely takes the previous definition of these values:

export const defaultRiskScoreBySeverity: Record<Severity, number> = {
  low: 21,
  medium: 47,
  high: 73,
  critical: 99,
};

and exposes them as constants for use in other files.

page: ALERTS_PAGE,
setting: STACK_BY_1_SETTING_NAME,
}),
plugin: APP_ID,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, I'd say let's default plugin APP_ID in the actual useLocalStorage. It can be in a follow up PR for sure if it'll be too much of a headache here with the tests 😂

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, I'd say let's default plugin APP_ID in the actual useLocalStorage. It can be in a follow up PR for sure if it'll be too much of a headache here with the tests 😂

done! 😄

Copy link
Member

@spong spong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked out, tested locally, and pair reviewed with @andrew-goldstein. Thank you for taking the time to go over everything here @andrew-goldstein, and for all your efforts in getting this amazing new alert triaging component in place! 🎉 This is going to greatly impact analysts alert triage experience, and is a step forward in optimizing and enhancing their workflow!

Overall everything in testing and review LGTM, so approving as-is. In testing we did come across a few items, so I'll note them below, but no need to address these as part of this PR, so feel free to either pickup in a follow-up, or create an issue for tracking/discussion as you see fit.

Welcome TreeMap! 🙌 LGTM! 🌲 🗺️ 🚀

Notes

Tree map isn't displayed for certain `group by top` fields, like `Ransomeware.version`, or `tags`
  • Consider a 'empty state' message similar to No data to display when both group by fields aren't supplied.

Extra left margins on `Table` view

Consider not refreshing/clearing data when deleting `group by` until focus is lost (or another selection is complete)
  • With regards to the group by selection UX, consider not updating data until a new field is selected, or focus is lost when empty (to prevent an unnecessary refresh when selecting a new field):

Allow `enter` to select initial item in group by components
  • When typing a field name we require an extra down-arrow + enter instead of just enter to confirm the first match (even if exact):

Consider mirroring same table/trend/tree component on Rule Details
  • I've always kinda paired the visualization above the Alerts table with the table itself, so it was interesting to see that we kept just the trend visualization on the Rule Details page. I think it would be an added benefit to mirror this new visualization UI on the Rule Details as well, so users have access to the treemap for additional grouping at the Rule layer. Would likely want it to have their own localstorage values and default group by's.

@spong
Copy link
Member

spong commented Jul 12, 2022

Side note: really loved the Color palette selection and Recommended groups features in the original POC. Hopefully we can enable these for 8.4 as well as they're fantastic QoL enhancements for this feature.

@@ -65,7 +68,6 @@ export const closeAlerts = () => {

export const expandFirstAlertActions = () => {
cy.get(TIMELINE_CONTEXT_MENU_BTN).should('be.visible');
cy.get(TIMELINE_CONTEXT_MENU_BTN).find('svg').should('have.attr', 'data-is-loaded');
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The should allow a user with crud privileges to attach alerts to cases test in attach_alert_to_case.spec.ts invokes the expandFirstAlertActions() test helper.

The test is failing on CI with the following error:

AssertionError: Timed out retrying after 120000ms: expected '<svg.euiIcon.euiButtonIcon__icon.css-e8xmlg-euiIcon-m-inherit>' to have attribute 'data-is-loaded'
    at expandFirstAlertActions (http://localhost:5620/__cypress/tests?p=cypress/integration/cases/attach_alert_to_case.spec.ts:22991:60)
    at Context.eval (http://localhost:5620/__cypress/tests?p=cypress/integration/cases/attach_alert_to_case.spec.ts:21750:50)

The expandFirstAlertActions() test helper in cypress/tasks/alerts.ts was updated ~ 21 hours ago with this change: https://github.com/elastic/kibana/pull/135373/files#diff-9881e669a4305a15a96f3880d5d68906edfc15991e384a71d7ef5498ed04c0c8R68hh

Based on desk testing, the data-is-loaded attribute:

Based on the git history, it looks like the expandFirstAlertActions() test helper didn't include this check:

cy.get(TIMELINE_CONTEXT_MENU_BTN).find('svg').should('have.attr', 'data-is-loaded');

in it's original implementation. The check was first added in this PR: 683463e#diff-9881e669a4305a15a96f3880d5d68906edfc15991e384a71d7ef5498ed04c0c8R68

Since this check only determines whether or not the SVG was loaded in the button icon, and data-is-loaded does not appear to be reilable, I'm removing this cy.get.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good 👍

I was the one that actually added that check, so for context, this is the comment that details the change: #130072 (comment)

When I ran into this I had no issue with it passing locally, but it would fail on CI, so you may have to click-loop and/or wait for the menu if there's still an issue waiting on the table to load before the icon is visible to click.

@@ -99,9 +99,9 @@ export const goToExceptionsTab = () => {
};

export const editException = () => {
cy.get(EXCEPTION_ITEM_ACTIONS_BUTTON).click();
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Displays version conflict error and Displays missing exception list error tests in exceptions_flyout.spec.ts invoke the editException() test helper.

This editException() helper function was added ~ 4 days ago, via this PR and is failing on CI with the following error:

CypressError: Timed out retrying after 120050ms: `cy.click()` failed because the center of this element is hidden from view:

`<button class="euiButtonIcon euiButtonIcon--primary euiButtonIcon--empty euiButtonIcon--xSmall" type="button" aria-label="Exception item actions menu" data-test-subj="exceptionItemCardHeader-actionButton">...</button>`

Fix this problem, or use `{force: true}` to disable error checking.

The test passes locally. I'm implementing the fix suggested in the error message above.

Copy link
Contributor

@michaelolo24 michaelolo24 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for doing this work and awesome job! LGTM 🚀

…ity-team#3221) to the Alerts page:

![alerts_by_risk_score_treemap](https://user-images.githubusercontent.com/4459398/156707243-cd9dc763-7d25-4de2-b53a-04139e334e01.png)

_Above: The new `Alerts by risk score` treemap_

- Alerts are colored by risk score
- Clicking on a cell instantly filters the alerts page
- Users may specify custom fields to group by, or quick-select from our curated sets of fields that pair well together

The features in this PR are enabled via the following feature flag in [experimental_features.ts](https://github.com/elastic/kibana/blob/main/x-pack/plugins/security_solution/common/experimental_features.ts):

```ts
  alertsTreemapEnabled: true,
```

When the feature flag is **disabled**, the alerts page looks like `main` today:

![feature_flag_disabled](https://user-images.githubusercontent.com/4459398/156687162-2be47b18-9413-4711-9f50-a9acb5904617.png)

_Above: This branch, when the feature flag is disabled_

When the feature flag is enabled, the layout of the Alerts page incorporates the design detailed in <elastic#120287>, which eliminates unusable space at the top of the Alerts page, and enables users to quickly hide and show the charts.

Users are introduced to the new `Hide charts` feature of the layout via `Step 1` of an [EUI Tour](https://elastic.github.io/eui/#/display/tour):

![tour_step_1](https://user-images.githubusercontent.com/4459398/156707574-9888290d-98e8-48a0-b920-4da6c56abcc5.png)

_Above: `Step 1` of the tour introduces the `Hide charts` feature_

Users may (optionally) try the `Hide charts` feature as part of the tour, per the video below:

<https://user-images.githubusercontent.com/4459398/156709026-ac16c97b-167e-4fce-b0e0-fd16077027b1.mov>

_Above: (video) The interactive tour let's users experience the new `Hide charts` feature_

`Step 2`, the final step of the tour, introduces the view selector:

![tour_step_2](https://user-images.githubusercontent.com/4459398/156707828-a1f742d8-3be2-4322-8bcf-dfd30a147670.png)

_Above: `Step 2` of the tour introduces the view selector_

Per the screenshot and video below, view selection toggles between the default `Trend view` and the new `Alerts by risk score view`:

![view_selection](https://user-images.githubusercontent.com/4459398/156708084-3c387887-ff2e-44f5-b729-7ed9f16e5297.png)

_Above: View selection_

<https://user-images.githubusercontent.com/4459398/156710378-3b9461c1-9779-4d66-8f12-21709d6f25fc.mov>

_Above: (video) View selection toggles between the  `Trend view` and the `Alerts by risk score view`_

We store the user's chart visibility and view preferences in local storage.

When users navigate to other views in Kibana or close the browser, their chart visibility and view preferences are restored the next time they visit the Alerts page.

Every rule, including prebuilt Elastic rules and custom rules created by users, must specify a risk score at rule creation time, per the screenshot below:

![rule_risk_score_configuration](https://user-images.githubusercontent.com/4459398/156712042-19b71f53-f337-4aed-bebf-ce10ea2b9f63.png)

_Above: Every rule has a risk score specified when it's created_

The colors of the alerts displayed in the treemap are determined by the rule's risk score.

This makes it easy to quickly filter the entire alerts page by clicking on the riskiest alerts, per the video below:

<https://user-images.githubusercontent.com/4459398/156714075-75112515-a247-4aa4-bc58-52fd17ce6fa0.mov>

_Above: (video) Users click on the riskiest alerts to filter the page_

Clicking on a cell in the treemap adds two filters, one for each group by field:

![two_filters](https://user-images.githubusercontent.com/4459398/156715601-5b7c0208-2fae-4231-ba3f-3073ef66cccf.png)

_Above: Two filters, (one for each group by field), are added to the page when a cell is clicked_

In the screenshot below, the 2nd filter was removed to reveal all the hosts in `mimikatz process started` alerts:

![2nd_filter_removed](https://user-images.githubusercontent.com/4459398/156716276-d582143a-a665-42f2-b1e2-122925d7399a.png)

_Above: Removing the 2nd filter, `host.name`, revealed all the hosts in the `mimikatz process started` alerts_

We can instantly answer questions like "Did this host trigger alerts for other rules?", and at a glance, grok the severity of those alerts, by removing the first of the two filters:

![1st_filter_removed](https://user-images.githubusercontent.com/4459398/156719888-51852cb5-c2bd-43b0-8a80-d1ee2c121bab.png)

_Above: Removing the first filter, `kibana.alert.rule.name`, revealed other alerts (from different rules) triggered for a specific host_

Users may customize the group by criteria:

<https://user-images.githubusercontent.com/4459398/156720911-8ed26b24-658f-4cc3-85b2-f317e15c430c.mov>

_Above: (video) Entering custom group by criteria_

When users navigate to other views in Kibana or close the browser, the previously-selected group by fields are restored from local storage the next time they visit the Alerts page.

Clicking on the first menu item in the settings gear, `Reset group by fields`, restores the default group by fields, per the video below:

<https://user-images.githubusercontent.com/4459398/156722389-2927d4e4-f8e8-4113-88d7-274a7c1e4bbe.mov>

_Above: (video) Restoring the default group by fields_

Users may quick-select from Elastic-curated sets of fields that pair well together, like parent and child processes, or process and file name:

![quick_select_groups](https://user-images.githubusercontent.com/4459398/156723673-cb405e41-9577-46a2-944b-44d4f1b2d6d7.png)

_Above: Quick-select from Elastic-curated sets of fiels_

Previous versions of the `Count` table were limited to grouping by one field. This PR enhances it to support grouping by two fields, per the screenshot below:

![count_two_group_by_fields](https://user-images.githubusercontent.com/4459398/156725762-1ea9ef4f-8d6c-4847-a637-f847df50bccd.png)

_Above: The `Count` table now supports two group by fields_

The group by criteria of the `Count` and `Alerts by risk score` widgets are linked together, making it easy to toggle between the two views, per the video below:

<https://user-images.githubusercontent.com/4459398/156727296-59fc1f29-e7a2-45eb-a549-5317100ab68a.mov>

_Above: (video) The `Count` and `Alerts by risk score` group by criteria are linked_

The `Trend` chart's legend has been enhanced to display counts, per the design detailed in issue <elastic#120282>

![counts_in_trend_widget_legend](https://user-images.githubusercontent.com/4459398/156728673-d365c88e-8da0-4cc4-a1cc-dbb5939a5c01.png)

_Above: The `Trend` chart legend, enhanced to display counts_

Note: The original intention of adding counts to the `Trend` chart legend, as documented in <elastic#120282>, was to combine the `Trend` and `Count` widgets into a single view via the legend.

In this PR, the `Trend` and `Count` widget queries have diverged; (the latter now supports multiple group by fields). This prevents the `Trend` and `Count` widgets from being combined into a single view.

This PR adds persistence of the `Trend` chart Stack by field to local storage.

When users navigate to other views in Kibana or close the browser, the previously-selected field is restored the next time they visit the Alerts page.
…uping

This PR introduces the new _Treemap_ and _Multi-Dimensional Alert Grouping_ to the Alerts page.

The initial commit was developed as an _ON week_ proof of concept (POC). It has since been updated to incorporate product and UX feedback.

### Alerts treemap

The new _Alerts_ page treemap is shown in the screenshot below:

![treemap](https://user-images.githubusercontent.com/4459398/178233664-c45be7ca-b03e-40b9-b423-aeeaa47461c0.png)

_Above: The new `treemap` in the Alerts page_

- Alerts are colored by risk score
- Clicking on a cell instantly filters the alerts page
- Treemap legend items may be added to filters and Timeline investigations
- The new treemap supports multi-dimensional grouping and filtering
  - Alerts are grouped by `kibana.alert.rule.name` and `host.name` by default

### Multi-Dimensional Alert Grouping

The table on the Alerts page, which previously supported grouping by a single field, has also been enhanced to support multi-dimensional grouping, per the screenshot below:

![alerts-table-multi-dimensional-grouping](https://user-images.githubusercontent.com/4459398/178240710-ecf66799-35a8-4874-8882-5ccfcccc86fe.png)

_Above: The table in the Alerts page, enhanced to support multi-dimensional grouping_

## Filtering the Alerts page by risk score

Every rule, including prebuilt Elastic rules and custom rules created by users, must specify a risk score at rule creation time, per the screenshot below:

![rule_risk_score_configuration](https://user-images.githubusercontent.com/4459398/156712042-19b71f53-f337-4aed-bebf-ce10ea2b9f63.png)

_Above: Every rule has a risk score specified when it's created_

The colors of the alerts displayed in the treemap are determined by the rule's risk score. This makes it easy to quickly filter the entire alerts page by clicking on the riskiest alerts.

Clicking on a cell in the treemap adds two filters, one for each group by field, per the screenshot below:

![two-filters](https://user-images.githubusercontent.com/4459398/178252768-7c66dc5e-8a3c-41d8-95e2-eeca20133127.png)

_Above: Two filters, (one for each group by field), are added to the page when a cell is clicked_

The Alerts page updates instantly when filters are added or removed. In the screenshot below, the 2nd filter was removed to filter the page to all `mimikatz process started` alerts:

![second-filter-removed](https://user-images.githubusercontent.com/4459398/178253726-66905d60-99da-4d76-9ea1-744cb53abd6f.png)

_Above: Removing the 2nd filter, a specific `host.name`, revealed all the hosts in the `mimikatz process started` alerts_

### Switching views

Users may switch between the following views:

- Table
- Trend (default)
- Treemap

per the screenshot below:

![view-selection](https://user-images.githubusercontent.com/4459398/178242001-6868c751-ffa6-486f-b81a-81a4d5912877.png)

_Above: View selection_

The default _Trend_ view is shown in the screenshot below:

![trend-view](https://user-images.githubusercontent.com/4459398/178242769-58d6c800-69db-4e14-87e4-9232f3e35427.png)

_Above: The (default) Trend view_

- The Trend chart's legend has been enhanced to display counts, per the design detailed in issue <elastic#120282>

- The Trend view only supports visualizing a single dimension. Hovering over the disabled `Group by top` select in the Trend view displays the tooltip shown in the screenshot below:

![tooltip](https://user-images.githubusercontent.com/4459398/178243356-9bfe7f54-5b31-4f61-a795-0cfa0a70285b.png)

_Above: The Trend view only supports visualizing a single dimension_

### Collapsing the panel

The panel may optionally be collapsed to save space, per the screenshot below:

![collapsed](https://user-images.githubusercontent.com/4459398/178244282-525d4c9f-a23b-4ec4-8f72-7e813e771687.png)

_Above: The panel (optionally) collapsed_

### User preferences are persisted to local storage

Previously, the _Group by_ selections on the Alerts page were always forgotten when users navigated away from the Alerts page. As a result, users had to re-select their preferred Group by fields every time they visited the page.

We now store all of the following preferences in local storage:

- View selection (Table, Trend, Treemap)
- Group by selections
- Panel collapse state

The preferences above are now restored the next time users return to the Alerts page.

### Group by field selection is synchronized between views

Group by field selection is synchronized between views. For example, if a user changes the Group by fields in the Treemap view and then switches to the Table view, the same Group by fields will be displayed in the Table view.

### Resetting Group by fields to their defaults

Users may reset the Group by fields to their defaults (`kibana.alert.rule.name` and `host.name`) for any visualization via the menu shown in the screenshot below:

![reset-group-by-fields](https://user-images.githubusercontent.com/4459398/178246997-70d89763-40d4-4c93-b0b6-b439fc3e22cd.png)

_Above: Resetting Group by fields to defaults via the menu_
…this PR

### `attach_alert_to_case.spec.ts`

The `should allow a user with crud privileges to attach alerts to cases` test in `attach_alert_to_case.spec.ts` invokes the `expandFirstAlertActions()` test helper.

The `expandFirstAlertActions()` test helper in `cypress/tasks/alerts.ts` was updated ~ 21 hours ago with this change: <https://github.com/elastic/kibana/pull/135373/files#diff-9881e669a4305a15a96f3880d5d68906edfc15991e384a71d7ef5498ed04c0c8R68hh>

Based on desk testing, the `data-is-loaded` attribute:
- Only indicates whether or not the button's SVG is loaded (it's an implementaiton detail of EUI: <https://github.com/elastic/eui/blob/b151438410eb39a39c4540e04810a5023530d77f/src/components/icon/icon.tsx>)
- Does not seem to be reliable: When inspecting the DOM via dev tools, the attribute is not present, but the icon is clearly visible / loaded

Based on the git history, it looks like the `expandFirstAlertActions()` test helper didn't include this check:

```ts
cy.get(TIMELINE_CONTEXT_MENU_BTN).find('svg').should('have.attr', 'data-is-loaded');
```

in it's original implementation. The check was first added in this PR: <elastic@683463e#diff-9881e669a4305a15a96f3880d5d68906edfc15991e384a71d7ef5498ed04c0c8R68>

Since this check only determines whether or not the SVG was loaded in the button icon, and `data-is-loaded` does not appear to be reilable, I'm removing this `cy.get`.

### `exceptions_flyout.spec.ts`

The `Displays version conflict error` and `Displays missing exception list error` tests in `exceptions_flyout.spec.ts` invoke the `editException()` test helper.

This `editException()` helper function was added ~ 4 days ago, [this PR](https://github.com/elastic/kibana/pull/135255/files#diff-e184b6d6539304e06fe6c7cdaa63773a60678e792d2c1b27a9873977c5a4e64bR101) and is failing on CI with the following error:

```
CypressError: Timed out retrying after 120050ms: `cy.click()` failed because the center of this element is hidden from view:

`<button class="euiButtonIcon euiButtonIcon--primary euiButtonIcon--empty euiButtonIcon--xSmall" type="button" aria-label="Exception item actions menu" data-test-subj="exceptionItemCardHeader-actionButton">...</button>`

Fix this problem, or use `{force: true}` to disable error checking.
```

The test passes locally. I'm implementing the fix suggested in the error message above.
@kibana-ci
Copy link
Collaborator

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] Security Solution Tests #3 / Inspect Hosts stats and tables "before all" hook for "inspects the All Hosts Table"

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
securitySolution 3118 3146 +28

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
securitySolution 5.2MB 5.2MB +22.1KB

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @andrew-goldstein

@andrew-goldstein andrew-goldstein merged commit 0301d20 into elastic:main Jul 13, 2022
@kibanamachine kibanamachine added the backport:skip This commit does not require backporting label Jul 13, 2022
@andrew-goldstein andrew-goldstein deleted the alerts-treemap branch July 13, 2022 04:25
andrew-goldstein added a commit to andrew-goldstein/kibana that referenced this pull request Aug 1, 2022
This PR contains fixes for the following issues:

- elastic#136377 was fixed by using the `key_as_string` value in the search strategy response to format timestamps for the Treemap and Table views, per the screenshots below:

**Treemap with timestamp formatting:**

![treemap_with_timestamp_formatting](https://user-images.githubusercontent.com/4459398/182256532-7df8d39b-4e09-440f-aae7-c6e3720e0906.png)

**Table with timestamp formatting:**

![table_with_timestamp_formatting](https://user-images.githubusercontent.com/4459398/182256817-ee8605a7-d9d3-4127-8753-aaad65832308.png)

- elastic#136387 was fixed by reducing the mininum font size to `4px` for group names, per [this example](https://elastic.github.io/elastic-charts/?path=/story/treemap--two-layers-stress-test&globals=theme:light;background:white) from Elastic charts. This change:
  - Reduces the chances of a label getting hidden when the browser is resized, but cannot eliminate it, because at a certain point, Elastic Charts must hide the label in order to (still) render the group
  - Will result in some groups having very small labels where the group previously would not have them, per the screenshot below:

![4px_labels](https://user-images.githubusercontent.com/4459398/182255505-bc77c184-df5a-4ff6-830a-c7c5b6127d25.png)

- elastic#136372 was fixed by adding the word `Trend`, `Table`, and `Treemap` to the inspect modal for each of the respective chart types, per the screenshots below:

**Inspect Trend:**

![inspect_trend](https://user-images.githubusercontent.com/4459398/182257336-5d988e6a-a436-48b6-ad59-72b64e86d89b.png)

**Inspect Table:**
![inspect_table](https://user-images.githubusercontent.com/4459398/182257396-6e0d01e3-e493-4765-ac60-fee790a52633.png)

**Inspect Treemap:**
![inspect_treemap](https://user-images.githubusercontent.com/4459398/182257457-d3b58f18-e393-4307-b319-a2e1019b0329.png)

- [review feedback](elastic#126896 (review)) where the `Tree map isn't displayed for certain group by top fields, like Ransomeware.version`, was fixed as suggested by displaying an empty state message with a reason why the treemap cannot be displayed:

**Before:**
![empty-state-before](https://user-images.githubusercontent.com/2946766/178567785-82d03c51-3e3c-49f6-ac86-d13ecaf3aef0.png)

**After:**
![empty-state-after](https://user-images.githubusercontent.com/4459398/182254376-437d3ce8-c4bd-4b43-a456-2a4460db4565.png)

- [review feedback](elastic#126896 (review)) where there were `Extra left margins when the Table view is selected` was fixed by removing the extra margin:

**Before:**
![left-margin-before](https://user-images.githubusercontent.com/2946766/178568473-c8162631-4340-4858-8d93-3466461f97d0.png)

**After:**
![left-margin-after](https://user-images.githubusercontent.com/4459398/182253474-1684094f-4f8d-43fe-860b-addb599ed4f1.png)
andrew-goldstein added a commit that referenced this pull request Aug 2, 2022
## [Security Solution] Alerts Treemap fixes

This PR contains fixes for the following issues:

- #136377 was fixed by using the `key_as_string` value in the search strategy response to format timestamps for the Treemap and Table views, per the screenshots below:

**Treemap with timestamp formatting:**

![treemap_with_timestamp_formatting](https://user-images.githubusercontent.com/4459398/182256532-7df8d39b-4e09-440f-aae7-c6e3720e0906.png)

**Table with timestamp formatting:**

![table_with_timestamp_formatting](https://user-images.githubusercontent.com/4459398/182256817-ee8605a7-d9d3-4127-8753-aaad65832308.png)

- #136387 was fixed by reducing the mininum font size to `4px` for group names, per [this example](https://elastic.github.io/elastic-charts/?path=/story/treemap--two-layers-stress-test&globals=theme:light;background:white) from Elastic charts. This change:
  - Reduces the chances of a label getting hidden when the browser is resized, but cannot eliminate it, because at a certain point, Elastic Charts must hide the label in order to (still) render the group
  - Will result in some groups having very small labels where the group previously would not have them, per the screenshot below:

![4px_labels](https://user-images.githubusercontent.com/4459398/182255505-bc77c184-df5a-4ff6-830a-c7c5b6127d25.png)

- #136372 was fixed by adding the word `Trend`, `Table`, and `Treemap` to the inspect modal for each of the respective chart types, per the screenshots below:

**Inspect Trend:**

![inspect_trend](https://user-images.githubusercontent.com/4459398/182257336-5d988e6a-a436-48b6-ad59-72b64e86d89b.png)

**Inspect Table:**
![inspect_table](https://user-images.githubusercontent.com/4459398/182257396-6e0d01e3-e493-4765-ac60-fee790a52633.png)

**Inspect Treemap:**
![inspect_treemap](https://user-images.githubusercontent.com/4459398/182257457-d3b58f18-e393-4307-b319-a2e1019b0329.png)

- [review feedback](#126896 (review)) where the `Tree map isn't displayed for certain group by top fields, like Ransomeware.version`, was fixed as suggested by displaying an empty state message with a reason why the treemap cannot be displayed:

**Before:**
![empty-state-before](https://user-images.githubusercontent.com/2946766/178567785-82d03c51-3e3c-49f6-ac86-d13ecaf3aef0.png)

**After:**
![empty-state-after](https://user-images.githubusercontent.com/4459398/182254376-437d3ce8-c4bd-4b43-a456-2a4460db4565.png)

- [review feedback](#126896 (review)) where there were `Extra left margins when the Table view is selected` was fixed by removing the extra margin:

**Before:**
![left-margin-before](https://user-images.githubusercontent.com/2946766/178568473-c8162631-4340-4858-8d93-3466461f97d0.png)

**After:**
![left-margin-after](https://user-images.githubusercontent.com/4459398/182253474-1684094f-4f8d-43fe-860b-addb599ed4f1.png)
kibanamachine pushed a commit that referenced this pull request Aug 2, 2022
## [Security Solution] Alerts Treemap fixes

This PR contains fixes for the following issues:

- #136377 was fixed by using the `key_as_string` value in the search strategy response to format timestamps for the Treemap and Table views, per the screenshots below:

**Treemap with timestamp formatting:**

![treemap_with_timestamp_formatting](https://user-images.githubusercontent.com/4459398/182256532-7df8d39b-4e09-440f-aae7-c6e3720e0906.png)

**Table with timestamp formatting:**

![table_with_timestamp_formatting](https://user-images.githubusercontent.com/4459398/182256817-ee8605a7-d9d3-4127-8753-aaad65832308.png)

- #136387 was fixed by reducing the mininum font size to `4px` for group names, per [this example](https://elastic.github.io/elastic-charts/?path=/story/treemap--two-layers-stress-test&globals=theme:light;background:white) from Elastic charts. This change:
  - Reduces the chances of a label getting hidden when the browser is resized, but cannot eliminate it, because at a certain point, Elastic Charts must hide the label in order to (still) render the group
  - Will result in some groups having very small labels where the group previously would not have them, per the screenshot below:

![4px_labels](https://user-images.githubusercontent.com/4459398/182255505-bc77c184-df5a-4ff6-830a-c7c5b6127d25.png)

- #136372 was fixed by adding the word `Trend`, `Table`, and `Treemap` to the inspect modal for each of the respective chart types, per the screenshots below:

**Inspect Trend:**

![inspect_trend](https://user-images.githubusercontent.com/4459398/182257336-5d988e6a-a436-48b6-ad59-72b64e86d89b.png)

**Inspect Table:**
![inspect_table](https://user-images.githubusercontent.com/4459398/182257396-6e0d01e3-e493-4765-ac60-fee790a52633.png)

**Inspect Treemap:**
![inspect_treemap](https://user-images.githubusercontent.com/4459398/182257457-d3b58f18-e393-4307-b319-a2e1019b0329.png)

- [review feedback](#126896 (review)) where the `Tree map isn't displayed for certain group by top fields, like Ransomeware.version`, was fixed as suggested by displaying an empty state message with a reason why the treemap cannot be displayed:

**Before:**
![empty-state-before](https://user-images.githubusercontent.com/2946766/178567785-82d03c51-3e3c-49f6-ac86-d13ecaf3aef0.png)

**After:**
![empty-state-after](https://user-images.githubusercontent.com/4459398/182254376-437d3ce8-c4bd-4b43-a456-2a4460db4565.png)

- [review feedback](#126896 (review)) where there were `Extra left margins when the Table view is selected` was fixed by removing the extra margin:

**Before:**
![left-margin-before](https://user-images.githubusercontent.com/2946766/178568473-c8162631-4340-4858-8d93-3466461f97d0.png)

**After:**
![left-margin-after](https://user-images.githubusercontent.com/4459398/182253474-1684094f-4f8d-43fe-860b-addb599ed4f1.png)

(cherry picked from commit a5a7d74)
kibanamachine added a commit that referenced this pull request Aug 2, 2022
## [Security Solution] Alerts Treemap fixes

This PR contains fixes for the following issues:

- #136377 was fixed by using the `key_as_string` value in the search strategy response to format timestamps for the Treemap and Table views, per the screenshots below:

**Treemap with timestamp formatting:**

![treemap_with_timestamp_formatting](https://user-images.githubusercontent.com/4459398/182256532-7df8d39b-4e09-440f-aae7-c6e3720e0906.png)

**Table with timestamp formatting:**

![table_with_timestamp_formatting](https://user-images.githubusercontent.com/4459398/182256817-ee8605a7-d9d3-4127-8753-aaad65832308.png)

- #136387 was fixed by reducing the mininum font size to `4px` for group names, per [this example](https://elastic.github.io/elastic-charts/?path=/story/treemap--two-layers-stress-test&globals=theme:light;background:white) from Elastic charts. This change:
  - Reduces the chances of a label getting hidden when the browser is resized, but cannot eliminate it, because at a certain point, Elastic Charts must hide the label in order to (still) render the group
  - Will result in some groups having very small labels where the group previously would not have them, per the screenshot below:

![4px_labels](https://user-images.githubusercontent.com/4459398/182255505-bc77c184-df5a-4ff6-830a-c7c5b6127d25.png)

- #136372 was fixed by adding the word `Trend`, `Table`, and `Treemap` to the inspect modal for each of the respective chart types, per the screenshots below:

**Inspect Trend:**

![inspect_trend](https://user-images.githubusercontent.com/4459398/182257336-5d988e6a-a436-48b6-ad59-72b64e86d89b.png)

**Inspect Table:**
![inspect_table](https://user-images.githubusercontent.com/4459398/182257396-6e0d01e3-e493-4765-ac60-fee790a52633.png)

**Inspect Treemap:**
![inspect_treemap](https://user-images.githubusercontent.com/4459398/182257457-d3b58f18-e393-4307-b319-a2e1019b0329.png)

- [review feedback](#126896 (review)) where the `Tree map isn't displayed for certain group by top fields, like Ransomeware.version`, was fixed as suggested by displaying an empty state message with a reason why the treemap cannot be displayed:

**Before:**
![empty-state-before](https://user-images.githubusercontent.com/2946766/178567785-82d03c51-3e3c-49f6-ac86-d13ecaf3aef0.png)

**After:**
![empty-state-after](https://user-images.githubusercontent.com/4459398/182254376-437d3ce8-c4bd-4b43-a456-2a4460db4565.png)

- [review feedback](#126896 (review)) where there were `Extra left margins when the Table view is selected` was fixed by removing the extra margin:

**Before:**
![left-margin-before](https://user-images.githubusercontent.com/2946766/178568473-c8162631-4340-4858-8d93-3466461f97d0.png)

**After:**
![left-margin-after](https://user-images.githubusercontent.com/4459398/182253474-1684094f-4f8d-43fe-860b-addb599ed4f1.png)

(cherry picked from commit a5a7d74)

Co-authored-by: Andrew Goldstein <[email protected]>
@tylersmalley tylersmalley added ci:cloud-deploy Create or update a Cloud deployment and removed ci:deploy-cloud labels Aug 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto-backport Deprecated - use backport:version if exact versions are needed backport:skip This commit does not require backporting ci:cloud-deploy Create or update a Cloud deployment release_note:enhancement Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Threat Hunting:Investigations Security Solution Investigations Team v8.4.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants